feat(state): encrypt secrets in state by default (auto local key, KMS on S3) - #1400

Re-implementation of #1030 (base had drifted to conflict), rebased onto current main. Opened by Claude (Fable) on behalf of @colelawrence.

State stores persist every Redacted<T> as a plaintext { "__redacted__": ... } marker. Secrets in state are now encrypted by default in the stores that own their key material, with nothing for the user to manage:

- { "apiKey": { "__redacted__": "sk-live-..." } }
+ { "apiKey": { "__secret__": "v1:kx4X0f..." } }

• Local (.alchemy/state/): auto-generated 32-byte key at ~/.alchemy/state.key (created on first use, mode 0600, wx-exclusive so concurrent creators converge). Repo state alone never exposes a secret.
• S3 (AWS.state()): KMS envelope encryption via the auto-managed alias/alchemy-state key, with the KMS-wrapped data key at {prefix}__state_key__.json. Engaged lazily — the codec resolves on write only when the value contains a Redacted, on read only when the raw JSON carries a __secret__ marker, so secret-free stacks never touch KMS (no kms:* permission, no CMK minted). Recovers the key from an out-of-band pending deletion. Opt out with secretEncryption: "off".
• ALCHEMY_PASSWORD overrides both with a scrypt-derived key.
• One codec (State/SecretCodec.ts): AES-256-GCM, random per-value IV, v1: framing. Sync by design (runs inside JSON.parse/stringify revivers); its own module so node:crypto stays out of workerd bundles (StateEncoding imports it type-only).
• Only Redacted payloads are ciphertext — the rest of state stays introspectable, and change detection is unaffected (secrets decrypt on read).
• Missing/wrong key fails with an actionable StateStoreError (shared stateDecodeError), never a defect or silent corruption.
• HTTP stores are untouched: shared by definition, at-rest protection is the server's job (the hosted store already encrypts server-side).

Rebase deltas vs. #1030

• StateEncoding.ts merged with the newer DATE_MARKER support; containsRedacted skips Date leaves.
• LocalState.ts merged with main's hardening (initialCwd anchoring, atomic temp-file writes, recoverMissingDir, deleteStack cache invalidation) — all preserved.
• alchemy state export (feat(cli): add bulk state read via alchemy state export #1043, post-dates feat(state): encrypt secrets in state by default (auto local key, KMS on S3) #1030) printed plaintext secrets; it now prints __secret__ envelopes via the same codec as state get:

  resources: exported.resources.map((r) => ({
    ...r,
-   state: encodeState(r.state),
+   state: encodeState(r.state, codec),
  })),

• PostgresState (feat(prisma): object storage (resources + bindings), locked Postgres state backend, Deployment.redeployOn #1061) writes rows client-side (unlike the HTTP store, the database never encrypts the JSON), but is shared across machines so it has no automatic key source. It now honors ALCHEMY_PASSWORD as an opt-in shared codec — encrypts on write, revives __secret__ envelopes on read (as typed StateStoreErrors, never defects). Without the password its behavior is unchanged.

Hardening from adversarial review

Three independent review passes (crypto, KMS/concurrency, integration) drove a hardening commit:

• Codec resolution memoizes on success only — Effect.cached persists failure Exits forever, so a transient S3/KMS/fs failure would have poisoned every later secret operation.
• recoverKmsKey rides out KMS eventual consistency (EnableKey retried through the post-CancelKeyDeletion window, then waits for KeyState=Enabled) instead of swallowing KMSInvalidStateException and leaving the key Disabled.
• The alias-race loser retries DescribeKey on NotFoundException and logs (instead of silently ignoring) a failed orphan-key cleanup; the IfNoneMatch put retries on 409 ConditionalRequestConflict per S3's documented semantics.
• Corrupt __state_key__.json / ~/.alchemy/state.key fail as typed StateStoreErrors, not JSON.parse/crypto defects; truncated v1: frames are rejected before reaching the cipher (Node accepts short GCM tags).
• The reviver only decrypts exact single-key { "__secret__": ... } envelopes — a user object merely containing the key is data. Reads engage the codec only when the raw JSON carries the marker, and a resolution failure surfaces only when a codec-less parse can't revive the state, so marker false-positives never demand KMS. Local writes touch the key file only when the value actually holds a Redacted; state get/export stay side-effect-free for secret-free output.

Known accepted trade-off: the scrypt salt is a fixed context string (documented in SecretCodec.ts) — ALCHEMY_PASSWORD is expected to be high-entropy (docs show openssl rand -base64 32), and a per-envelope salt would require a v2: frame format.

Compatibility

• Old plaintext __redacted__ markers still revive; the next write re-encrypts (explicit migration tests, local and S3).
• One-way format migration: alchemy versions predating __secret__ read the envelope as a plain object instead of failing. Readers of a shared store must upgrade together — called out in a :::caution in the docs; should be in the release notes too.

Tested: 14 hermetic encoding/keyfile/migration tests, plus live S3 tests covering the KMS flow end-to-end (raw-object ciphertext assertion, legacy roll-forward without engaging KMS, secret-free-never-touches-KMS, pending-deletion recovery).

Docs: rewritten "Secrets in state are encrypted" section on environments/secrets.

🤖 Generated with Claude Code

Rebase onto main (maintainer)

Rebased onto current main with three follow-up commits:

• rootDir is now rootDir() from Auth/Paths.ts (ALCHEMY_HOME-aware); SecretCodec is deep-imported and not re-exported from State/index.ts, so node:crypto stays out of worker bundles.
• The alchemy state get/export changes were dropped: those commands were replaced by alchemy state read (feat!: overhaul the Alchemy CLI #1235), which decrypts through the store like any other reader.
• Added test/State/SecretEncryptionCompat.test.ts (local store: keyless legacy reads never create a key, mixed legacy/encrypted stages, lost-key and wrong-password failure modes, concurrent key creation, forward-compat envelope shape) and Postgres compatibility tests in test/State/PostgresState.test.ts.
• Docs: a lost machine key fails reads with a StateStoreError (it is not "re-written on redeploy"); teams sharing .alchemy/state/ or reading it from CI must set ALCHEMY_PASSWORD before upgrading.